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An image processing apparatus includes 
a designating section for designating an arbitrary 
region or an arbitrary position of an image; a 
specifying section for specifying an object region 
which is present in the designated region or position, 
and which can additionally be in a vicinity of the 
designated region or position, from pixel information 
in the designated region or position; a determining 
section for determining an image region to be cut out 
from the image, based on the specified object region; 
and a cutting section for cutting out the 
determined image region from the image. 
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ABSTRACT OF THE DISCLOSURE 

An image processing apparatus includes a 
designating section for designating an arbitrary region 
5 or an arbitrary position of an image; a specifying 
section for specifying an object region which is present 
in the designated region or position, and which can 
additionally be in a vicinity of the designated region or 
position, from pixel information in the designated region 
10 or position; a determining section for determining an 
image region to be cut out from the image , based on the 
specified object region; and a cutting section for 
cutting out the determined image region from the image - 
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BACKGROUND OF THE INVENTION 

1. FIELD OF THE INVENTION: 

The present invention relates to an image pro- 
5 cessing apparatus for use in computers for conducting 
image processing, word processors, portable information 
tools, copying machines, scanners, facsimiles or the 
like. More specifically, the present invention relates 
to an image processing apparatus enabling a user to 

10 designate the coordinates of any point on the image by a 
coordinate input apparatus such as a mouse, a pen or a 
tablet, or an image processing apparatus capable of 
photoelectrical ly converting a printed image on a piece 
of paper or the like with coordinates being designated in 

15 a different type of ink so as to input the image and the 
coordinates, wherein the image processing apparatus being 
capable of cutting out an object image with an arbitrary 
size at an arbitrary position from the original image. 

20 2- DESCRIPTION OF THE RELATED ART: 

When an image including an object or a person's 
face of interest is cut out from the original image, the 
image is cut with a desired size using a pair of scis- 
sors, a cutter or the like, in the case of a photograph. 
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In the case of an electronic image obtained by a CCD 
camera or a scanner, however, the positions of two points 
are designated by a coordinate input device such as a 
mouse, using software for image processing (e.g., the 
5 image processing software "PhotoShop" made by Adobe 
Inc.)/ and a rectangle having a diagonal between the two 
points is designated as a region. 

In order to output a part of the original image, 
10 which includes an object of interest, as an image having 
a particular size, a portion having the object of inter- 
est at a well-balanced position is first cut out from the 
original image , and thereafter, is magnified /reduced to 
a required size. In the case of a photograph, such 
15 magnification/reduction is conducted by, for example, a 
copying machine. In the case of an electronic image, 
magnifying/ reducing the image to a desired size can be 
easily carried out. However, cutting out a portion 
having the object of interest at a well-balanced position 
20 must be conducted before such magnification/reduction. 

Furthermore, in order to extract a region repre- 
senting a person's face except for hair (hereinafter, 
this portion is referred to as a "face skin") from the 
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original image, a face skin region which is visually 
determined by an operator is painted out. In the case of 
an electronic image, a pixel is designated by a coordi- 
nate input device such as a mouse, and those pixels 
5 having a similar color to that of the designated pixel 
are combined to be extracted as one region ( e . g * , 
"PhotoShop" as mentioned above). There is also a method 
as follows: the color distribution of a face skin is 
analyzed in advance to set a probability density func- 

10 tion. Then, the probability density of the input pixels 
is obtained using values such as RGB (red, green, blue) 
values and HSV (hue, color saturation, brightness) values 
as arguments, thereby designating those pixels having a 
probability equal to or higher than a prescribed value as 

15 a face- skin region (R. Funayama, N. Yokoya, H. Iwasa and 
H. Takemura, "Facial Component Extraction by Cooperative 
Active Nets with Global Constraints", Proceedings of 13th 
International Conference on Pattern Recognition, Vol. 2, 
pp. 300-305, 1996). 

20 

Conventionally, in the case where a rectangle 
including a face-skin region in the image is determined, 
the rectangle is commonly determined visually by an 
operator. 
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Moreover / the central axis of a person's face has 
been commonly detected based on the visual determination 
of an operator, 

5 Another method for detecting the central axis of 

the face is as follows: a skin-color portion of the face 
is extracted as a region, and the region is projected to 
obtain a histogram. Then, the right and left ends of the 
face are determined from the histogram, whereby the line 
10 passing through the center thereof is determined as the 
central axis of the face (Japanese Laid-Open Publication 
No. 7-181012). 

Furthermore, respective vertical positions of the 
15 nose, the eyes and the mouth on the face have been 
commonly detected based on the visual determination of an 
operator . 

Another method is to match an image template of 
20 the nose with an input image ( *Face Recognition: Features 
versus Templates*, by R. Brunelli and T. Poggio, IEEE 
Transactions on Pattern Analysis and Machine Intelli- 
gence, Vol.15, No. 10, pp. 1042-1052, 1993). In this 
article, a method for detecting the vertical positions by 
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projecting a gray-level image or an edge Image to obtain 
a histogram, and examining peaks and valleys of the 
histogram, has also been proposed. 

5 Moreover, the width of the face has been commonly 

detected based on the visual determination of an opera- 
tor. 

Another method is as follows: a skin-color 
10 portion of the face is extracted as a region, and the 
region is projected to obtain a histogram. Then, the 
right and left ends of the face are determined from the 
histogram, whereby the distance between the ends is 
obtained as the width of the face ( Japanese Laid-open 
15 Publication No. 7-181012). 

As described above, in order to output a part of 
the original image, which includes a person's face of 
interest, as an image having a particular size, a portion 
20 having the face at a well-balanced position is first cut 
out from the original image, and thereafter, is magni- 
fied/reduced to a required size. In the case of a 
photograph, such magnification/reduction is conducted by, 
for example, a copying machine. In the case of an 
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electronic image, magnifying/reducing -the image to a 
desired size can be carried out easily. However, cutting 
out a portion having the object of interest at a well- 
balanced position must be conducted before such 
magnification/reduction . 

In the case of an electronic image, it is also 
possible for a user to adjust, in advance, the size of 
the face of the original image to an appropriate size, 
move a frame on the screen according to the visual 
determination of the user so that the face is located in 
the center, and output the image located within the 
frame. An apparatus achieving such an operation has been 
proposed in Japanese Laid-Open Publication No. 64-82854. 

In order to achieve improved visual recognition 
of a person's face on a photograph or an image, the 
amount of exposure light for printing is adjusted in the 
case of a photograph. For an electronic image, there is 
software capable of conducting adjustment of contrast, 
tonality and brightness, edge sharpening, blurring 
processing and the like (e.g., "PhotoShop" as mentioned 
above ) . 
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When an image including an object or a person's 
face of interest is cut out from the original image , the 
image is cut with a desired size using a pair of scis- 
sors, a cutter or the like, in the case of a photograph. 
5 However, using a pair of scissors, a cutter or the like 
to cut an image is actually time-consuming. Moreover, 
cutting a portion including the object or the face of 
interest at a well-balanced position requires much skill. 
When software for processing an electronic image obtained 

10 by a CCD camera or converted by a scanner is utilized 
(e.g. f "PhotoShop" as mentioned above), the positions of 
two points are usually designated by a coordinate input 
device such as a mouse, and a rectangle having a diagonal 
between the two points is designated as a region. In 

15 this case as well, cutting out a portion including an 
object or a face of interest at a well-balanced position 
requires much skill. Furthermore, in the case where an 
object or a face of interest is originally located at the 
edge of the screen, and a portion including the object or 

20 the face at a well-balanced position in the center is to 
be cut out from the image, it is necessary to first cut 
out the portion from the original image, and thereafter, 
move the position of the object or the face to the center 
of the resultant image. 
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As described above, in order to output a part of 
the original image , which includes an object of interest, 
as an image having a particular size, a portion having 
the object of interest at a well-balanced position is 
5 first cut out from the original image, and thereafter, is 
magnified/reduced to a required size. In the case of a 
photograph, such magnification/reduction is conducted by, 
for example, a copying machine. However, the image is 
not always cut to the same size. Therefore, in order to 

10 obtain an image with a desired size, a troublesome 
operation of calculating the magnification/ reduction 
ratio is required. In the case of an electronic image, 
magnifying/reducing the image to a desired size is easy. 
However, cutting out a portion having the object of 

15 interest at a well-balanced position must be conducted 
before such magnification/reduction. In short, at least 
two operations are required to output an image having a 
particular size. 

20 Furthermore, the above-mentioned method of 

painting out a visually determined face-skin region is 
troublesome regardless of whether an image to be pro- 
cessed is a photograph or an electronic image. Moreover, 
painting a portion at the boundary between the face skin 
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region and the other regions must be conducted extremely 
carefully. In the case of an electronic image, the 
above-mentioned method of combining those pixels having 
similar color to that of the designated pixel to extract 
5 them as one region (e.g. "PhotoShop" as mentioned above) 
has been used. In this method, however, since the colors 
of the skin, the lip and the eyes are different, it is 
necessary to combine the results of several operations in 
order to extract the whole face- skin. Moreover, the skin 

10 color may be significantly uneven even in the same person 
due to , for example , di f f er en t skin shades or shadows . 
In this case as well, the results of several operations 
must be combined. Also described above is the method of 
designating those pixels having a probability equal to or 

15 higher than a prescribed value as a face-skin region (the 
above-cited reference by R. Funayama, N. Yokoya, H. Iwasa 
and H. Takemura). According to this method, however, a 
face- skin region might not be successfully extracted in 
the case where the image's brightness is extremely uneven 

20 due to the photographing conditions or the conditions at 
the time of obtaining the image, or in the case where the 
color of the skin is different due to a racial differ- 
ence . 
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As described above, when a rectangle including a 
f ace - skin region is to be obtained , the rectangle has 
been commonly determined visually by an operator. 
However, such a method is troublesome regardless of 
5 whether an image to be processed is a photograph or an 
electronic image. 

Moreover, in the above-mentioned method of 
detecting the central axis of a person T s face from a 
10 histogram (Japanese Laid-Open Publication No. 7-181012), 
the correct central axis can only be detected in the case 
where the face is completely directed to the front, while 
the correct central axis can not be obtained in the case 
where the face is turned even slightly to either side. 

15 

Furthermore, according to the above-mentioned 
method of matching an image template of the nose with an 
input image (the above-cited reference by R. Brunelli and 
T. Poggio), it is desirable that the size of the nose to 
20 be extracted is known. In the case where the size of the 
nose is not known, templates of various sizes must be 
matched with the input image, requiring substantial time 
for calculation. Moreover, according to the above- 
mentioned method of detecting the vertical positions by 
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examining peaks and valleys of the histogram ( the above- 
cited reference by R. Brunelli and T. Poggio), the 
vertical positions might not be correctly extracted, for 
example, in the case where the face skin region or the 
5 background is not known. In short, wrong extraction 
could occur without precondition. 

Moreover, according to the above-mentioned method 
to detect a width of the face (Japanese Laid-Open Publi- 

10 cation No. 7-181012), a face skin region should be cor- 
rectly extracted based on the color information. Howev- 
er, in the case where a background region includes a 
color similar to that of the face skin, a region other 
than the face skin region might be determined as a face 

15 skin, or a shaded portion in the face skin region might 
not be determined as face skin. The detected width of 
the face might be different depending upon whether or not 
the ears can be seen on the image. Moreover, the detect- 
ed width could be larger than the correct width in the 

20 case where the face is turned toward either side. 

As described above, in order to output a part of 
the original image, which includes an object of interest, 
as an image having a particular size, a portion having 
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the object of interest at a well-balanced position is 
first cut out from the image , and thereafter, is magni- 
fied/reduced to a required size. In the case of a 
photograph, such magnification/reduction is conducted by, 
5 for example, a copying machine. However, the image is 
not always cut to the same size. Therefore, in order to 
obtain an image with a desired size, a troublesome 
operation of calculating the magnification/reduction 
ratio is required. In the case of an electronic image, 

10 magnifying/reducing the image to a desired size can be 
easily carried out. However, cutting out a portion 
having the object of interest at a well-balanced position 
must be conducted before such magnification/reduction. 
In short, at least two operations are required to output 

15 an image having a particular size. According to a 
somewhat automated method as described in Japanese Laid- 
Open Publication No. 64-82854, the user adjusts, in 
advance, the size of the face of the original image to an 
appropriate size, moves a frame on the screen according 

20 to the visual determination of the user so that the face 
is located in the center, and output the image located 
within the frame. Alternatively, the user adjusts, in 
advance, the size of the face of the original image to an 
appropriate size, moves a T-shaped indicator on the 
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screen according to the visual determination of the user 
so that the ends of the horizontal line of the T-shaped 
indicator overlap the eyes, respectively, and then, 
outputs an image within a rectangle defined with an 
5 appropriate margin from the T-shaped indicator. 

The above-described operation of adjusting the 
amount of exposure light for printing in order to achieve 
improved visual recognition of a person's face on a 

10 photograph or an image requires much skill. For an elec- 
tronic image, there is software capable of conducting 
adjustment of contrast, tonality and brightness, edge 
sharpening, blurring processing and the like (e.g., 
"PhotoShop" as mentioned above), as described above. In 

15 this case as well, using such software requires much 
skill, and usually, various operations must be tried 
until a desired image is obtained. 



SUMMARY OF THE INVENTION 

20 

According to one aspect of the present invention, 
an image processing apparatus includes a designating 
section for enabling a user to designate an 
arbitrary image region or an arbitrary image 
25 position without designating a contour of an object 
region which is present in the user designated 
image region or position; a specifying section for 
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specifying the object region which is present in the user 
designated image region or position, and which can 
additionally be in a vicinity of the user designated image 
region or position, from pixel information in the user 
designated image region or position; a determining section 
for determining an image region to be cut out from the 
image, based on the specified object region and pixel 
information which can additionally be in the vicinity of 
the user designated image region; and a cutting section 
for cutting out the determined image region from the 
image . 

In one example, the determining section includes 
a section for adjusting a size of the image region to a 
prescribed size. 

In one example, the determining section includes^ 
a correcting section for entirely correcting the user 
designated image region or correcting only a part of the 
user designated image region. 

According to another aspect of the present 
invention, an image processing apparatus includes a 
designating section for enabling a user to designate an 
arbitrary image region or an arbitrary image without 
designating a contour of a face image region which is 
present in the user designated image region or position; 
an analyzing section for analyzing a color distribution in 
the designated region or position and in a vicinity of the 
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designated region or position; an adjusting section for 
adjusting a condition for specifying the face image which 
is present in the image, according to a result of the 
analysis; a specifying section for specifying the face 
5 image region which is present in the user designated 
image region or position, and which can additionally 
be in the vicinity of the user designated image region 
or position, based on the adjusted condition; a 
determining section for determining an image region to be 
10 cut out from the image, based on the specified face image 
region; and a cutting section for cutting out the 
determined image region from the image. 

In one example, the determining section includes 
15 a section for adjusting a size of the image region, using 
the user designated image region or the user designated 
image position as a reference. 

In one example, the specifying section includes 
20 a section for applying noise elimination or labelling to 
the specified face image region to produce a face mask; 
a section for vertically scanning the produced face mask 
to obtain a sum of vertical differential luminance values 
of pixels in the image corresponding to the face mask to 
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produce a histogram; and a section for detecting a 
central axis of a face from a profile of the produced 
histogram. 

5 In one example, the specifying section includes 

a section for applying noise elimination or labelling to 
the specified face image region to produce a face mask; 
a section for vertically scanning the produced face mask 
to obtain a mean luminance value of pixels in the image 
10 corresponding to the face mask to produce a histogram ; 

and a section for detecting a vertical nose position from 
a profile of the produced histogram. 

In one example, the specifying section includes 
15 a section for applying noise elimination or labelling to 
the specified face image region to produce a face mask; 
a section for horizontally scanning the produced face 
mask to obtain a mean luminance value of pixels in the 
image corresponding to the face mask to produce a histo- 
20 gram; and a section for detecting a vertical eye position 
from a profile of the produced histogram. 

In one example, the specifying section includes 
a section for applying noise elimination or labelling to 
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the specified face image region to produce a face mask; 
a section for horizontally scanning the produced face 
mask to obtain a mean luminance value of pixels in the 
image corresponding to the face mask to produce a histo- 
5 gram; and a section for detecting a vertical mouth 
position from a profile of the produced histogram. 

In one example, the specifying section further 
includes a section for detecting a vertical eye position 
10 from the profile of the produced histogram; and a section 
for obtaining a middle position of a region between the 
detected vertical eye position and the detected vertical 
mouth position to detect a width of the face mask at the 
middle position. 

15 

In one example, the determining section includes 
a section for adjusting a position of the image region, 
based on the face image region, a central axis of a face 
in the face image, a vertical nose position of the face 
20 in the face image, a vertical eye position of the face in 
the face image, a vertical mouth position of the face in 
the face image, and a width of a face mask of the face 
image. 
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In one example, the determining section includes 
a section for adjusting a size of the image region, based 
on the face image region, a central axis of a face in the 
face image, a vertical nose position of the face in the 
face image, a vertical eye position of the face in the 
face image, a vertical mouth position of the face in the 
face image, and a width of a face mask of the face image. 

In one example, the determining section includes 
a correcting section for entirely correcting the user 
designated image region or correcting only a part of. 
the user designated image region. 

Thus, the invention described herein makes 
possible -the advantage of providing an image processing 
apparatus capable of photoelectrically converting a 
printed image on a piece of paper or the like with 
coordinates being designated in a different type of ink 
so as to input the image and the coordinates, wherein the 
image processing apparatus being capable of cutting out 
an object image with an arbitrary size at an arbitrary 
position from the original image. 

This and other advantages of the present inven- 
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tion will become apparent to those skilled in the art 
upon reading and understanding the following detailed 
description with reference to the accompanying figures. 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram showing an image 
processing apparatus according to one example of the 
present invention; 

10 

Figure 2 is a block diagram showing an im- 
age/coordinate input apparatus in the image processing 
apparatus shown in Figure 1; 

15 Figure 3 is a block diagram showing another im- 

age/coordinate input apparatus in the image processing 
apparatus shown in Figure 1; 

Figure 4 shows examples of a region of an object 
20 or a face in the image designated by the user; 

Figure 5 shows examples of a position of the 
object or a face in the image designated by the user; 



BNSDOCID: <CA 2240745C__I_> 



CA 02240745 1998-06-16 



S98099 

- 20 - 

Figures 6A through 6D show images illustrating 
the steps from the user's designation to the extraction 
of an image; 

5 Figure 7 is a flow chart illustrating Image 

processing procedure 1 conducted by the image processing 
apparatus of the example shown in Figure 1; 

Figure 8 is a diagram showing the pixels of an 
10 object region; 

Figure 9 is a diagram illustrating how a part of 
an image is attached to a document; 

15 Figure 10 is a diagram illustrating an example of 

extracting only a face-skin portion from the image 
including a person's face; 

Figures 11A, 11B and 11C show the frequency 
20 histograms plotted with respect to the hue, color 
saturation and brightness, respectively; 

Figure 12 is a flow chart illustrating Image 
processing procedure 3 for producing an image represent - 
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ing a face skin region; 

Figures 13A and 13B show input patterns designat- 
ed by the user; 

Figure 14A shows an example of the image; 



Figure 14B is a graph showing the relationship 
between brightness and frequency of the image of Figure 
10 14A; 

Figure 15 is a diagram illustrating an example of 
extracting only a face skin portion from the image 
including a person's face; 

15 

Figure 16 is a diagram illustrating how the size 
of a window region is gradually increased; 

Figure 17 shows an input pattern designated by 
20 the user; 

Figure 18 is a flow chart illustrating Image 
processing procedure 5 for producing a face mask by the 
image processing apparatus of the example shown in Figure 
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1; 

Figure 19 illustrates how the face mask is 
produced; 

5 

Figure 20 is a flow chart illustrating the 
process for detecting the central axis of the face; 

Figure 21 is a diagram illustrating the process 
10 for detecting the central axis of the face; 

Figure 22 is a flow chart illustrating Image 
processing procedure 6 for detecting a vertical position 
of the nose by the image processing apparatus of the 
15 example shown in Figure 1; 

Figure 23 is a diagram illustrating the process 
for detecting the vertical position of the nose; 

20 Figure 24 is a flow chart illustrating Image 

processing procedure 7 for detecting a vertical position 
of the eyes by the image processing apparatus of the 
example shown in Figure 1; 



BNSDOCID: <CA 2240745C__I_> 



CA 02240745 1998-06- 16 



S98099 

- 23 - 

Figure 25 is a diagram illustrating the process 
for detecting the vertical position of the eyes; 

Figure 26 is a flow chart illustrating Image 
5 processing procedure 8 for detecting a vertical position 
of the mouth by the image processing apparatus of the 
example shown in Figure 1; 

Figure 27 is a diagram illustrating the process 
10 for detecting the vertical position of the mouth; 

Figure 28 is a flow chart illustrating Image 
processing procedure 9 for detecting a width of a face 
mask by the image processing apparatus of the example 
15 shown in Figure 1; 

Figure 29 is a diagram illustrating the process 
for detecting the width of the face mask; 

20 Figure 30 is a flow chart illustrating Image 

processing procedure 10 for cutting out a rectangular 
image from the original image by the image processing 
apparatus of the example shown in Figure 1; 
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Figure 31 shows a sheet of an address book with 
a face image being attached thereto; and 

Figure 32 is a diagram illustrating the process 
for correcting an image. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 



Hereinafter, the present invention will be 
10 described by way of illustrative examples with reference 
to the accompanying drawings. The same reference numer- 
als designate the same component. 

Figure 1 is a block diagram showing an image 
15 processing apparatus according to one example of the 
present invention. An image to be processed and coordi- 
nates required for the processing are input by an im- 
age/coordinate input apparatus 1-1. In the case where 
the image is in a digital form, the image is directly 
20 stored in an input image storing section 1-2-1 of a 
storage apparatus 1-2. In the case where the input image 
is an analog form, the image is converted into a digital 
form, and the resultant image is stored in the input 
image storing section 1-2-1. The input coordinates are 
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stored in an input coordinate storing section 1-2-2. An 
image processing section 1^3 uses the stored image and 
coordinates as input information to conduct an appropri- 
ate image processing in an operation region of a memory 
5 within the image processing section 1-3. Thereafter, the 
image processing section 1-3 stores the resultant image 
and coordinates in an output image storing section 1-2-3 
and an output coordinate storing section 1-2-4 of the 
storage apparatus 1-2, respectively. After undergoing 
10 processing, the resultant image can be sent to an image 
output apparatus 1-4, whereby a copy of the resultant 
Image can be made. 

Figures 2 and 3 are diagrams illustrating in 
15 detail the image/coordinate input apparatus 1-1 shown in 
Figure 1 . 

The image/coordinate input apparatus 1-1 in 
Figure 1 separately includes an image input apparatus 2-1 
20 and a coordinate input apparatus 2-2, as shown in Figure 
2. The input image from the image input apparatus 2-1 is 
stored in the input image storing section 1-2-1 of the 
storage apparatus 1-2, whereas the input coordinates from 
the coordinate input apparatus 2-2 are stored in the 
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input coordinate storing section 1-2-2 of the storage 
apparatus 1-2. For example, a camera capable of directly 
inputting a digitized image by a solid-state image 
sensing device (CCD; charge coupled device); an apparatus 
5 capable of digitizing a photograph or a scanner which can 
input printed matters; or an apparatus for holding a 
digitized image such as equipment connected to a network, 
like the internet, and a magnetic storage apparatus may 
be used as the image input apparatus 2-1. As the 
10 coordinate input apparatus 2-2, a mouse capable of 
inputting coordinates with a pointer displayed on a 
display, a track ball, a pen-type apparatus, a pen- type 
coordinate input apparatus using a tablet, a coordinate 
input apparatus using a finger, or the like may be used. 

15 

The image/coordinate input apparatus 1-1 in 
Figure 1 includes an image reading apparatus 2-3 and an 
image/coordinate separation apparatus 2-4, as shown in 
Figure 3- This type of the image/coordinate input 
20 apparatus 1-1 is used in the case where both an image 
including an object to be processed and input coordinates 
are present on a single image. For example, in the case 
where a line or a point representing the coordinates is 
drawn in a particular color on a photograph, only a 
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component of that color is extracted to obtain a separate 
image. Thereafter, the position of the point or the line 
is analyzed from the separate image, whereby the coordi- 
nates are extracted. 

5 

Figure 4 shows examples of a region of the object 
in the image designated by the user. First, an image and 
a pattern indicated by a solid line or points, as shown 
in Figure 4, are input to the image/coordinate input 

10 apparatus 1-1 (Figure 1). In the case of a rectangular 
pattern 4-1, the coordinates of two points, that is, the 
coordinates of the upper left point and the lower right 
point of the pattern are used as the input coordinates* 
In the case of a pattern 4-4, 4-10, 4-11, 4-12, 4-13 or 

15 4-14, the coordinates of the upper left point and the 
lower right point of a rectangle circumscribing the input 
pattern (i.e., such a rectangle as shown by a dotted line 
on each image) are used as the input coordinates. In the 
case of the other patterns, two coordinates defining a 

20 rectangle circumscribing the input pattern can be used as 
the input coordinates. However, in the case of a line or 
dot pattern, that is, in the case of a pattern 4-2, 4-3, 
4-5 or 4-6, no rectangle circumscribing the pattern could 
be obtained. Otherwise, such a rectangle that has an 
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extremely large aspect ratio would be obtained. In such 
a case, an appropriate rectangle will be set according to 
a mean aspect ratio of the object (this rectangle will be 
set as a square when the object is not known). In the 
5 case of a pattern 4-2, for example, the object is a 
person's face and a rectangle circumscribing the input 
pattern is extremely long in the longitudinal direction 
(or the input pattern is a vertical straight line and no 
rectangle circumscribing the input pattern can be ob- 

10 tained). In such a case, a rectangle as shown by a 
dotted line is set. In other words, a rectangle horizon- 
tally magnified/reduced from the rectangle circumscribing 
the input pattern is obtained by multiplying the length 
of the rectangle circumscribing the input pattern by a 

15 prescribed ratio. Furthermore, the coordinates of the 
upper left point and the lower right point are used as 
the input coordinates. In the case of a pattern 4-7, 4-8 
or 4-9, a rectangle longitudinally and laterally magni- 
fied from the rectangle circumscribing the input pattern 

20 by respective prescribed ratios is set, and the coordi- 
nates of two points of the rectangle are used as the 
input coordinates . 

Figure 5 shows examples of a position of the 



BNSOOCID: <CA 2240745C_I_> 



CA 02240745 1998-06-16 

S98099 

- 29 - 

object designated by the user. In the case where the 
user designates a point such as a pattern 5-1, the 
coordinates of that point can be used as the input 
coordinates. In the case where the user designates a 
5 pattern other than the point such as a pattern 5-2, the 
center of a circumscribed rectangle can be used as the 
input coordinates . 

[Image processing procedure 1] 
10 Image processing procedure 1 conducted by the 

image processing apparatus of the present example will 
now be described with reference to the flow chart of 
Figure 7. 

15 First, using the image/coordinate input apparatus 

1-1 (Figure 1), the user roughly designates a region of 
the object in the image stored in the input image storing 
section 1-2-1, as shown in Figure 4, or roughly desig- 
nates a position of the object, as shown in Figure 5. 

20 Figures 6A through 6D show images illustrating the steps 

from the user's designation to the extraction of an 
image. When a region 6-1-1 is designated by the user 
(Step Sl-1), as shown in Figure 6A, the image processing 
section 1-3 obtains a rectangular region 6-1-2 reduced 
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from the rectangle circumscribing the input pattern by an 
appropriate ratio, and stores the region 6-1-2 as a set 
region in the input coordinate storing section 1-2-2 
(Step Sl-7)- As shown in Figure 6B, when a position 6-2- 
1 is designated by the user (Step Sl-2), the image pro- 
cessing section 1-3 obtains an appropriate rectangular 
region 6-2-2 centered around the designated position 6-2- 
1 (Step Sl-3), and stores the region 6-2-2 in the input 
coordinate storing section 1-2-2 (Step Sl-7). 

The image processing section 1-3 (Figure 1) 
utilizes the operation region of the memory within the 
image processing section 1-3 to store the color informa- 
tion of the pixels included in the rectangular region 6- 
1-2 or 6-2-2 (Step Sl-4), and sets the rectangular region 
6-1-2 or 6-2-2 as an initial value of the object region 
(Step Sl-5). 

Figure 8 shows the pixels in the object region. 
The image processing section 1-3 finds a pixel 8-2 
adjacent to the object region 8-1. When the pixel 8-2 
satisfies at least one of the following two conditions 
(Step Sl-6), the pixel 8-2 is added to the object region 
(Step Sl-9): 
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1. the color difference between the pixel of 
interest and an adjacent pixel in the object region is 
within a prescribed range; and/or 

5 2. the color difference between the pixel of 

interest and a pixel stored in Step Sl-4 is within a 
prescribed range* 

The image processing section 1-3 examines all of 
10 the pixels adjacent to the object region in terms of the 
above two conditions. This operation is repeated until 
no pixel can be added to the object region. Then, as 
shown in Figure 6C, the image processing section 1-3 
obtains a final object region 6-3-1 (Step Sl-8). It 
15 should be noted that, although various indices of the 
color difference have been proposed, a Godlove's color- 
difference formula as shown in "Improved Color-Difference 
Formula with Applications to the Perceptibility and 
Acceptability of Fadings" , I. H. Godlove, J. Opt. Soc. 
20 Am., 41, 11, pp. 760-772, 1951 may be used. 

The image processing section 1-3 expresses the 
area of the object region 6-3-1 as the number of pixels 
included in the object region 6-3-1. Then, as shown in 
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Figure 6D, the image processing section 1-3 obtains a 
rectangular region 6-3-3 centered around the center of 
gravity of the object region 6-3-1 and having an area 
corresponding to a prescribed percentage (e.g., 30%) of 
5 the total area of the rectangular region 6-3-3. Thereaf- 
ter, the image, processing section 1-3 cuts out the 
rectangular region 6-3-3 from the original image. The 
shape of the rectangular region 6-3-3 may be square. 
Alternatively, the shape of the rectangular region 6-3-3 

10 may be set as appropriate depending upon applications. 

For example, the rectangular region 6-3-3 may be set to 
have a ratio of 4:3 according to the aspect ratio of a 
television screen, or may be set to have a ratio of 16:9 
according to the aspect ratio of a high-definition 

15 television screen. It should be noted that, although the 
rectangular region is centered around the center of 
gravity of the object region in the above description, 
the position of the center of gravity in the rectangular 
region may be shifted longitudinally and laterally 

20 depending upon the application. 

A method for obtaining the center of gravity is 
described in, for example, "Robot Vision" by M. Yachida, 
Shohkohdo, ISBN4-7856-3074-4 C3355, 1990. A part of the 
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image can be cut out from the original image, based on 
the coordinates of the rectangular region. 



[Image processing procedure 2] 
5 The image processing section 1-3 magnifies or 

reduces the image which has been cut out according to 
Image processing procedure 1, to an appropriate size, and 
stores the resultant image in the output image storing 
section 1-2-3 of the storage apparatus 1-2. The image 

10 processing section 1-3 may utilize the stored image for 
any appropriate applications. For example, an image 9-1 
including an automobile and obtained by a digital camera, 
as shown in Figure 9, is stored in the input image 
storing section 1-2-1. Then, a part of the image 

15 including only the automobile is cut out from the input 

image. Thereafter, this part of the image is attached to 
a report 9-2 having a prescribed format and a frame for 
a prescribed image size. The resultant report 9-2 is 
stored in the output image storing section 1-2-3. 

20 

[Image processing procedure 3] 

Before Image processing procedure 3, the color 
distribution of a person's face skin is analyzed in 
advance according to the following procedures: 
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1, the face-skin portion is manually extracted 
from a face image 10-1 to produce a face-skin image 10-2 
(Figure 10); 

5 2. a face-skin image is similarly produced for a 

plurality of different persons; 

3. frequency histograms are plotted with respect 
to the hue (Figure 11A, 11-1-1), color saturation (Figure 

10 HB, 11-2-1) and brightness (Figure 11C, 11-3-1) of the 

pixels of each of the face-skin images to obtain the 
color distribution; and 

4. for each histogram, the mean and variance of 
15 the distribution are obtained, and such a normal 

probability density function (11-1-2, 11-2-2, 11-3-2) 
that best fits the distribution is obtained. 

Thus, the color distribution of the face skin can 
20 be expressed by the normal probability density functions 
(P hue (hue), P sat (sat) and P val (val)) of the hue, color 
saturation and brightness, each function having two argu- 
ments: the mean and variance (u hue , o 2 hue ; p sat , o 2 aat ; and 
Hvai' ° 2 vai/ respectively). In this specification, each of 
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the normal probability density functions is referred to 
as a skin-region probability density function. Each 
skin-region probability density function is expressed by 
the following expressions: 

5 

P hue (hue) - N(p hue , o 2 hue ) -.-(1) 
F sat (sat) - N(p sat , a 2 saC ) *..(2) 
P val (val) - N(u val , a 2 , al ) -..(3) 

10 When the calculated mean and variance are applied 

to the normal distribution, those values which are 
significantly different from a mean value, if any, would 
result in a greater estimation of the variance than the 
actual variance. Even a few values would cause such an 

15 estimation. For example, in the case of the hue distri- 
bution histogram as shown in Figure 11A, most of the 
pixels are distributed within about ±30 of about 20. In 
this histogram, values such as 100 and -150 would result 
in a grater estimation of the variance. Therefore, in 

20 order to obtain a normal distribution curve (a probabili- 
ty density function) which can be applied to a more 
accurate distribution, it would be better to first remove 
those pixels having such values, and thereafter, calcu- 
late the mean and variance. 
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The image processing section 1-3 stores each of 
the normal probability density functions in advance, and 
processes the image stored in the input image storing 
section 1-2-1 according to the flow chart of Figure 12. 
5 In Step Sl-0, the image processing section 1-3 sets an 
original processing region, based on the user input. In 
the case where a pattern (region) 9-1 as shown in Figure 
13A is input from the image/ coordinate input apparatus 1- 
1 to the input coordinate storing section 1-2-2, *the 

10 image processing section 1-3 sets a processing region 9-2 
of the image stored in the input image storing section 1- 
2-1 in such a way as described above- In the case where 
a pattern (position) 9-4 as shown in Figure 13B is input, 
the image processing section 1-3 sets a processing region 

15 9-5 (Step Sl-0). The image processing section 1-3 
substitutes a hue value, a color-saturation value and a 
brightness value of each pixel in the respective normal 
probability density functions obtained as described 
above, so as to obtain the respective probabilities. 

20 Such a pixel that has a value equal to or higher than a 
prescribed probability with respect to each of the hue, 
color saturation and brightness is determined as an 
original probable face-skin pixel (Step S2-1). At this 
time, the prescribed probability should be set to a small 
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value such as 5% so that as many pixels as possible may 
be selected as a probable face-skin pixel* Thus, any 
pixels which possibly correspond to the face-skin portion 
are determined as original probable face-skin pixels, 
5 Thereafter, the image processing section 1-3 calculates 
the mean and variance of each of the hue, color satura- 
tion and brightness (Step S2-2). In the foregoing de- 
scription, an original probable face-skin pixel is 
selected based on the probabilities of the hue, color 
10 saturation and brightness* However, it may also be 
effective to adjust each threshold to a value close to 
the pixel value of the above-mentioned prescribed 
probability, depending upon the characteristics of an 
imaging system . 

15 

Provided that the mean and variance of the hue, 
color distribution and brightness thus calculated are p 
'hue. <* 2 'hu e '- M'sat, o 2 \«z'' ^d p\ al , o 2 * val , respectively, 
corresponding probability density functions P' hue (hue), 
20 P , oac (sat) and P' val (val) having these arguments can be 
expressed by the following expressions: 

P' hue (hue) ~ N(p' hue , a 2 * hue ) ...(4) 
P' sat (sat) - N(p' aat , a 2 sat ) -..(5) 
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P' val (val) - N(u' val , a 2 ' val ) ...(6) 

Using these probability density function, the 
image processing section 1-3 selects face-skin pixels 
5 according to the following procedures: 

1. first, all of the pixels in the image are set 
as initial values, and any pixels having a value equal to 
or lower than a prescribed probability (P ! hue (hue)) calcu- 

10 lated from a hue value as an argument are removed (Step 
S2-3); 

2. next, any pixels having a value equal to or 
lower than a prescribed probability (P' aat (sat)) calculat- 

15 ed from a color- saturation value as an argument are 
removed (Step S2-4); and 

3. finally, any pixels having a value equal to or 
lower than a prescribed probability (P' val (val)) calculat- 

20 ed from a brightness value as an argument are removed 
(Step S2-5). 

As a result, a face-skin region is specified 
(Step S2-6). 
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The lower limit of each probability is set higher 
than they were set when the original probable face-skin 
pixels were obtained. For example, provided that the 
previous threshold of the probability is 5% as described 
5 above, the threshold may be set to 30%. As a result, 
more accurate extraction can be carried out. More 
specifically, any pixels that have been wrongly extracted 
as not being noise based on the 5% threshold, would be 
removed based on the 30% threshold. 

10 

In the foregoing description, selection of the 
pixels corresponding to the face-skin portion is conduct- 
ed based on the probabilities. However, it may also be 
effective to adjust each threshold to a value close to 

15 the pixel value of the above-mentioned prescribed 
probability, depending upon the characteristics of an 
imaging system. For example, as can be seen from Figure 
14A, the face skin and the hair of an image 14-1 have 
different brightnesses. Figure 14B is a histogram 

20 showing the brightness versus frequency of the image of 
Figure 14A. As shown in Figure 14B, a peak 14-2 repre- 
senting the hair appears at a lower value of the bright- 
ness, whereas a peak 14-3 representing the face-skin 
region appears at a relatively higher value of the 
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brightness. Provided that a peak value is simply select- 
ed as a threshold of the brightness probability of the 
image 14-1, the peak value 14-2 might be set as a thresh- 
old, whereby those pixels corresponding to a part of the 
5 hair might be selected as the pixels corresponding to the 
face skin. In such a case, such an algorithm as an 
Ohtsu's discriminant analysis method (which is described 
in the above-cited reference: "Robot Vision" by M. 
Yachida) may be applied to a value equal to or lower than 
10 an appropriate brightness value to set a more appropriate 

value 14-5 as the brightness threshold. 

By updating the skin region probability density 
functions as appropriate in such a manner as described 
15 above, an image 12-3 representing a face-skin region can 
be obtained from an image 12-1, as shown in Figure 15 
(Step S2-6). The image 12-3 thus obtained has a smaller 
amount of noise, as compared to an image 12-2 conven- 
tionally extracted using a fixed function. 

20 

[Image processing procedure 4] 

Image processing procedure 4 is conducted after 
the image representing the face skin-region is obtained 
according to Image processing procedure 3. Referring to 
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an image 16-1 in Figure 16, in -the case where only a 
position 16-1-0 is designated by the user, the image 
processing section 1-3 sets the smallest rectangle 16-1-1 
centered around the designated point, and sets a region 
5 16-1-3 located between the rectangle 16-1-1 and a slight- 
ly larger rectangle 16-1-2 as an initial window region. 
The image processing section 1-3 gradually magnifies the 
window region 16-1-3 as shown in images 16-2 and 16-3, 
until one of the four sides of the outer rectangle 16-1-2 

10 of the window region 16-1-3 reaches the edge of the input 
image. Thereafter, the image processing section 1-3 
calculates the dispersion of the pixels of the window 
region 16-1-3 in the image representing the face-skin 
region. The largest dispersion will be calculated when 

15 both the face skin and the contour of a part other than 
the face skin appear in the window region as shown in an 
image 16-4. Accordingly, during the operation of gradu- 
ally magnifying the window region 16-1-3, t;he image 
processing section 1-3 determines the outer rectangle 16- 

20 1-2 corresponding to the largest dispersion, as a rectan- 
gle including the face skin region. 

As shown in Figure 17, in the case where a region 
15-1, not a position, is designated by the user, the 
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image processing section 1-3 magnifies or reduces an 
outer rectangular defining a window region by an 
appropriate ratio to the size smaller than that of a 
rectangle 15-2 obtained from the designated region 15-1. 
5 Thus, the smallest rectangle 15-3 is obtained, whereby an 
initial window region is set such that the outer rectan- 
gle defining the window region corresponds to the rectan- 
gle 15-3. Thereafter, the image processing section 1-3 
gradually magnifies the window region, until an inner 

10 rectangle of the window region becomes lager than a 
rectangle 15-4 magnified by an appropriate ratio from the 
rectangle 15-2. The image processing section 1-3 then 
calculates the dispersion of the pixels within the window 
region in a similar manner, and determines the outer 

15 rectangle corresponding to the largest dispersion, as a 
rectangle including the face-skin region. It should be 
noted that, provided that the region designated by the 
user is only slightly shifted from the face region, the 
rectangle magnified by an appropriate ratio from the 

20 rectangle obtained from the designated region may be 
determined as the rectangle including the face- skin 
region. 

[Image processing procedure 5] 
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Figure 18 is a flow chart showing Image process- 
ing procedure 5 conducted by the image processing section 
1-3. The image processing section 1-3 processes an input 
color image 17-1 shown in Figure 19 according to Image 
5 processing procedure 4 to obtain a rectangle including a 
face-skin region 17-2, The image processing section 1-3 
processes that rectangle according to Image processing 
procedure 3 to obtain an image 17-3 representing a face 
skin region as shown in Figure 19. The image processing 

10 section 1-3 combines the pixels connected to each other 
in the face-skin region image 17-3 to produce a label 
image. The image processing section 1-3 then extracts 
only a label region having the largest area from the 
produced label image, and forms a binary image 17-4 from 

15 the label region (Step S3-1). Regarding the image 17-4, 
the image processing section 1-3 replaces black pixels 
(holes) surrounded by white pixels with white pixels to 
fill the holes. As a result, an image 17-5 is formed 
(Step S3-2). The image processing section 1-3 first 

20 reduces the size of the image 17-5 once (Step S3-3 ) , and 
again produces a label image. The image processing 
section 1-3 extracts only a label region having the 
largest area from the label image (Step S3-4). After 
magnifying the resultant image n times (Step S3-5), the 



BNSDOCID: <CA 2240745C__L> 



CA 02240745 1998-06-16 

S98099 

- 44 - 

image processing section 1-3 reduces the size of the 
image n times (Step S3-6), and extracts only a label 
region having the largest area from the resultant label 
image (Step S3-7). Thus, a face mask 17-6 is obtained. 

In the above steps, n should be set to, for 
example, 3 or 4 depending upon the size, characteristics 
or the like of the image. The magnifying and reducing 
processing as described above is described in the 
above-cited reference: "Robot Vision" by M. Yachida. 

The face mask 17-6 thus obtained is used to 
define the range to be subjected to the processing 
according to the flow chart shown in Figure 20. The 
15 image processing section 1-3 extracts only luminance 
components from the input color image 17-1 to obtain a 
gray-level image 17-2 (Step S4-1 ) . At the same time, the 
image processing section 1-3 produces the face mask 17-6 
according to the flow chart in Figure 18 (Step S3-0 ) . 
20 The image processing section 1-3 differentiates the gray- 
level image 17-2 in a vertical direction with respect to 
the white pixels in the face mask 17-6 to obtain a 
differentiated image 17-7 (Step S4-2 ) . In the image 
17-7, those pixels corresponding to the black pixels in 



5 



10 
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the face mask 17-6 are set to zero. Such a differentiat- 
ed image is commonly obtained by using, for example, a 
Prewitt's operator (the above-cited reference; "Robot 
Vision" by M. Yachida). 

5 

The image processing section 1-3 projects the 
image 17-7 in a vertical direction to obtain a histogram 
17-8 (Step S4-3). A vertical axis of the histogram 17-8 
shows the sum of the pixel values of the image 17-7 at a 

10 corresponding horizontal position. Referring to Figure 
21, the image processing section 1-3 sets such a vertical 
axis 21-la that horizontally divides the histogram 21-1 
into two regions: right and left • regions . The image 
processing section 1-3 obtains such an axis 21-2 that has 

15 the smallest value of SSDS given by the following expres- 
sion: 

(a-i)>a mai and (a+iVca^ 

SSDS= £{(/(a-0-/(a+0) 2 } 

1=1 

20 where a indicates a position of the axis 21-la, a nin 
indicates a left end of the histogram, a^ indicates a 
right end of the histogram, and f(s) indicates a height 
of the histogram (Step S4-4). Then, the image processing 
section 1-3 sets the position 21-2 as a central axis 21-3 
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of the face. 



[Image processing procedure 6] 

Figure 22 is a flow chart illustrating Image 
processing procedure 6 performed by the image processing 
section 1-3- The image processing section 1-3 produces 
the gray-level image 17-2 and the face mask 17-6 based on 
the image 17-1 as shown in Figure 23 (Steps S4-1 and S3- 
0). The image processing section 1-3 horizontally scans 
only the gray-level image within the face mask 17-6 to 
produce a histogram 18-1 projecting a mean luminance 
value (Step S5-1). The image processing section 1-3 then 
produces a histogram 18-2 having a reduced resolution 
from the histogram 18-1 (Step S5-2), and searches for a 
peak position 18-2-1 approximately in the middle of the 
lower-resolution histogram 18-2 (Step S5-3). In the case 
where no peak is found (Step S5-6, No), the image 
processing section 1-3 sets the position in the middle of 
the histogram as a vertical nose position (Step S5-5). 
In the case where any peak is found (Step S5-6, Yes), the 
image processing section 1-3 scans a region around the 
position of the histogram 18-1 corresponding to the 
detected peak of the lower- resolution histogram 18-2, in 
order to search for a peak position 18-3-1 (Step S5-4). 
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The image processing section 1-3 sets this peak position 
18-3-1 as the vertical nose position (Step S5-0). 

[Image processing procedure 7] 
5 Figure 24 is a flow chart illustrating Image 

processing procedure 7 conducted by the image processing 
section 1-3. The image processing section 1-3 produces 
a horizontal histogram 25-5 as shown in Figure 25 accord- 
ing to Image processing procedure 6 (Step S5-10). Using 

10 this histogram 25-5, the image processing section 1-3 

scans a region 25-1 above a vertical nose position 25-6 
detected in Image processing procedure 6 to detect the 
deepest two valleys 25-2 and 25-3 (Step S6-1). In the 
case where the two valleys are both detected (Step S6-3), 

15 the image processing section 1-3 sets the lower one of 
the valleys , that is, the valley 25-3 as a vertical 
position 25-7 of the eyes (Step S6-2). In the case where 
only one valley is detected (Step S6-4), the image 
processing section 1-3 sets the detected valley as the 

20 vertical eye position (Step S6-5). In the case where no 
valley is detected, the image processing section 1-3 sets 
the position in the middle of the region between the 
vertical nose position and the upper end of the histogram 
25-5 as the vertical eye position (Step S6-6). 
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[Image processing procedure 8] 

Figure 26 is a flow chart illustrating Image 
processing procedure 8 conducted by the image processing 
section 1-3. The image processing section 1-3 produces 
5 a horizontal histogram 26-1 as shown in Figure 27 accord- 
ing to Image processing procedure 6 (Step S5-10). Using 
the histogram 26-1, the image processing section 1-3 
scans a region 26-3 below the vertical nose position 26-2 
detected in Image processing procedure 6 to detect the 

10 deepest three valleys 26-4, 26-5 and 26-6 (Step S7-1). 

In the case where the three valleys are detected (Step 
S7-2), the image processing section 1-3 sets the middle 
one of the valleys, that is, the valley 26-5 as a verti- 
cal position 26-7 of the mouth (Step S7-5), as shown in 

15 an image 26-8. 

In the case where only two valleys are detected 
(Step S7-3), the image processing section 1-3 first 
detects the widths of a face mask 26-11 at the two 
20 valleys. Then, the image processing section 1-3 calcu- 
lates the ratio of the width 26-10 of the face mask 26-11 
at the lower valley to the width 26-9 at the upper 
valley. In the case where the calculated ratio is higher 
than a prescribed value (e.g., 0-7) (Step S7-6), the 
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image processing section 1-3 sets the position of the 
upper valley as a vertical mouth position (Step S7-9). 
Otherwise, the image processing section 1-3 sets the 
position of the lower valley as the vertical mouth 
5 position (Step S7-10). 

In the case where only one valley is detected 
(Step S7-4), the image processing section 1-3 sets the 
position of the detected valley as the vertical mouth 
10 position (Step S7-7). 

In the case where no valley is detected / the 
image processing section 1-3 sets the position in the 
middle of the region between the vertical nose position 
15 and the lower end of the histogram 26-1 as the vertical 
mouth position (Step S7-8). 

[Image processing procedure 9] 

Figure 28 is a flow chart illustrating Image 
20 processing procedure 9 conducted by the image processing 
section 1-3. As shown in Figure 29, a face mask 28-1, a 
vertical eye position 28-2 and a vertical mouth position 
28-3 are obtained according to Image processing proce- 
dures 7 and 8 (Steps S3-0, S6-0 and S7-0). The image 
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processing section 1-3 horizontally scans the pixels from 
the vertical eye position 28-2 to the vertical mouth 
position 28-3 in order to obtain a width of the face mask 
28-1. The image processing section 1-3 obtains a width 
5 in the middle of the region between the vertical posi- 
tions 28-2 and 28-3 as a width 28-4 of the face (Step 
S29-1) - 

[image processing procedure 10] 

10 Figure 30 is a flow chart illustrating Image 

processing procedure 10 conducted by the image processing 
section 3-1. the face mask, the central axis of the 
face, the vertical eye position, the vertical mouth 
position, and the width of the face are detected accord- 

15 ing to Image processing procedures 5, 6, 7, 8 and 9. The 
distance between the eyes and the mouth can be obtained 
from the vertical eye position and the vertical mouth 
position. Using such information, the image processing 
section 1-3 cuts out an image which includes a face 

20 having an appropriate size and located at a well-balanced 
position in the horizontal and vertical directions, from 
the original image. 

First, the image processing section 1-3 deter- 
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mines whether or not the detected width of the face is 
reliable. The width of the face is detected according to 
Image processing procedure 9, and the central axis of the 
face is detected according to Image processing procedure 
5 5. Accordingly, the width of the face is divided into 
two widths by the central axis. A width on the left side 
of the central axis is herein referred to as a left-face 
width , whereas a width on the right side of the central 
axis is herein referred to as a right- face width. The 

10 image processing section 1-3 verifies that the left- face 
width and the right-face width are not zero (Step S10-1). 
Then, the image processing section 1-3 calculates the 
ratio of the left-face width to the right-face width to 
determine whether or not the calculated ratio is within 

15 a prescribed threshold- range (Step S10-2). In the case 
where the ratio is not within the threshold-range ( Step 
S10-2, Yes), the image processing section 1-3 determines 
that the detected width of the face is not reliable, and 
determines a rectangle to be cut out from the detected 

20 eye-mouth distance (Step S10-6). More specifically, the 
image processing section 1-3 sets the intersection of the 
central axis of the face and the vertical nose position 
as a reference point. Then, the image processing section 
1-3 calculates a rectangle centered around the reference 
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point and having a width and length each calculated as a 
product of the eye-mouth distance and a respective 
prescribed ratio (Step S10-6). Thus, the rectangle to be 
cut out is obtained. 

5 

In the case where the width of the face is 
reliable (Step S10-2, No), the image processing section 
1-3 determines whether or not the detected eye-mouth 
distance is reliable (Step S10-3 ) . The image processing 

10 section 1-3 calculates the ratio of the detected eye- 
mouth distance to the length of the detected rectangle 
circumscribing a pattern designated by the user, and 
determines whether or not the calculated ratio is within 
a prescribed threshold-range (Step S10-3, No)- Note that 

15 in the case where a position, not a region, is designated 
by the user, the image processing section 1-3 calculates 
the ratio of the detected eye-mouth distance to a rectan- 
gle reduced by a prescribed ratio from the face-skin 
region obtained according to Image processing procedure 

20 4. In the case where the ratio is not within the thresh- 
old-range, the image processing section 1-3 determines 
that the detected vertical eye position and the detected 
vertical mouth position (and the detected eye-mouth 
distance) are not reliable, and determines a rectangle to 
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be cut out from the detected width of the face. More 
specifically, the image processing section 1-3 sets as a 
reference point the intersection of the detected central 
axis of the face and the vertical center line of the 
5 rectangle circumscribing the pattern designated by the 
user. Then, the image processing section 1-3 calculates 
a rectangle centered around the reference point and 
having a width and length each calculated as a product of 
the width of the face and a respective prescribed ratio. 
10 Thus, the rectangle to be cut out is obtained (Step S10- 
5). 

In the case where both the width of the face and 
the eye-mouth distance are reliable (Step S10-3, Yes), 

15 the image processing section 1-3 determines a rectangle 
to be cut out from these two values. More specifically, 
the image processing section 1-3 sets the intersection of 
the detected central axis of the face and the vertical 
nose position as a reference point, and calculates 

20 weighted arithmetic mean values by respectively 
multiplying the width of the face and the eye-mouth 
distance by a prescribed ratio. Then, the image process- 
ing section 1-3 calculates a rectangle centered around 
the reference point and having a width and length each 
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calculated as a product of the respective calculated 
arithmetic mean value and a respective prescribed ratio 
(Step S10-4). Thus, a rectangle to be cut out is ob- 
tained . 

5 

Finally, the image processing section 1-3 calcu- 
lates the ratio of the size of the rectangle thus ob- 
tained to the size of the rectangle circumscribing the 
pattern designated by the user, and determines whether or 

10 not the calculated ratio is within a prescribed thresh- 
old-range (Step S10-7). In the case where the ratio is 
not within the threshold- range, the image processing 
section 1-3 determines that the obtained rectangle is not 
appropriate, and determines a rectangle from the pattern 

15 designated by the user. More specifically, in the case 
where a region is designated by the user, the image 
processing section 1-3 sets the center of a rectangle 
circumscribing the region as a reference point. Then, 
the image processing section 1-3 calculates a rectangle 

20 centered around the reference point and having a width 
and length each calculated as a product of the length of 
the circumscribing and a respective prescribed ratio 
(Step S10-8). Thus, the rectangle to be cut off is 
obtained. In the case where a position is designated by 
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the user, the center of the rectangle including a face 
skin region obtained according to Image processing 
procedure 4 is used as a reference point, and similar 
processing is carried out to obtain a rectangle to be cut 
5 out. 

[Image processing procedure 11] 

The image processing section 1-3 magnifies or 
reduces the face image which is cut out according to 

10 Image processing procedure 10 to an appropriate size, and 
stores the resultant image in the output image storing 
section 1-2-3 of the storage apparatus 1-2. The image 
processing section 1-3 can utilize the stored face image 
for appropriate applications such as an address book in 

15 a portable information tool. For example, the image 
processing section 1-3 stores an image of a person ob- 
tained by a digital camera, such as an image 30-1 as 
shown in Figure 31, in the input image storing section 1- 
2-1, and roughly designates a portion in and around the 

20 face using the image coordinate input apparatus 1-1. 

Then, the image processing section 1-3 cuts out an image 
including the face at a well-balanced position from the 
original image according to Image processing procedure 
10, and magnifies or reduces the resultant image to fit 
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a prescribed frame. Thus, the resultant image is at- 
tached to a document, as shown by an image 30-2 of Figure 
31. The image 30-2 is a sheet of the address book with 
the face image being attached thereto. 

5 

[Image processing procedure 12] 

A face mask is obtained according to Image 
processing procedure 12. In order to improve the visual 
recognition of the face in the image, the image process- 

10 ing section 1-3 of the present example appropriately pro- 
cesses only a portion of the input image corresponding to 
a white-pixel region of the face mask to make the image 
characteristics of the face-skin region and the other 
regions different from each other. Alternatively, in 

15 order to improve the visual recognition of the face in 
the image, the image processing section 1-3 may appropri- 
ately process only a portion of the input image corre- 
sponding to a black-pixel region of the face mask to make 
the image characteristics of the face region and the 

20 other regions different from each other. 

For example, Figure 32 is a diagram illustrating 
the image correction processing. In the case where a 
face mask 31-2 is obtained from an input image 31-1, the 
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image processing section 1-3 reduces the sharpness of the 
portion of the input image corresponding to the black- 
pixel region of the face mask 31-2, using a Gaussian 
filter or an averaging filter. As a result, an image 31- 
5 3 having reduced visual recognition of the background 
other than the face and having improved visual recogni- 
tion of the face is obtained. In the case where the 
input image is not a sharp image, the image processing 
section 1-3 improves the visual recognition of the face 

10 by processing the portion of the input image correspond- 
ing to the white-pixel region of the face mask by, for 
example, edge sharpening. As a result, an image 31-4 is 
obtained. Similar effects may be obtained by reducing 
the contrast of the image, instead of reducing the 

15 sharpness of the regions other than the face region. In 
the case where the input image is a low-contrast image, 
similar effects may be obtained by increasing the con- 
trast of the face- skin region. Alternatively, the 
contrast of the whole input image may be increased so 

20 that the portion of the input image corresponding to the 
white-pixel region of the face mask has the highest 
contrast . 

According to the present invention, the user 
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roughly designates a position (or a region) of the object 
in the original image, whereby an image which includes 
the object at a well-balanced position can be cut out 
from the original image - 

5 

In one example, the user roughly designates a 
position (or a region) of the object in the original 
image, whereby an image having a prescribed size and 
including the object at a well-balanced position can be 
10 output. 

In one example, the user roughly designates a 
position (or a region) of the person's face in the image, 
whereby a region representing a face skin can be extract- 
15 ed. 

In one example, the user roughly designates a 
position (or a region) of the person's face in the image, 
whereby a rectangle including a region representing the 
20 face skin can be obtained. 

In one example, the user roughly designates a 
position (or a region) of the person's face in the image, 
whereby the central axis of the face can be detected. 



BNSDOCID: <CA 2240745C_I_> 



CA 02240745 1998-06-16 

S98099 

- 59 - 

In one example, the user roughly designates a 
position (or a region) of the person's face in the image, 
whereby a vertical position of the nose in the face can 
be detected. 

5 

In one example, the user roughly designates a 
position (or a region) of the person's face in the image, 
whereby a vertical position of the eyes in the face can 
be detected. 

10 

In one example, the user roughly designates a 
position (or a region) of the person's face in the image, 
whereby a vertical position of the mouth in the face can 
be detected. 

15 

In one example, the user roughly designates a 
position (or a region) of the person's face in the image, 
whereby a width of the face can be detected. 

20 In one example, the user roughly designates a 

position (or a region) of the person's face in the 
original image, whereby an image which includes the face 
at a well-balanced position can be cut out from the 
original image. 
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In one example, the user roughly designates a 
position (or a region) of the person's face in the image, 
whereby an image having a prescribed size and including 
the face at a well-balanced position can be output . 

In one example, the user roughly designates a 
position (or a region) of the person's face in the image, 
whereby the image quality can be adjusted so that the 
visual recognition of the face is improved. 

Various other modifications will be apparent to 
and can be readily made by those skilled in the art 
without departing from the scope and spirit of this 
invention. Accordingly, it is not intended that the 
scope of the claims appended hereto be limited to the 
description as set forth herein, but rather that the 
claims be broadly construed. 
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THE EMBODIMENTS OF THE INVENTION IN WHICH AN EXCLUSIVE PROPERTY 
OR PRIVILEGE IS CLAIMED ARE DEFINED AS FOLLOWS: 

1. An image processing apparatus, comprising: 

a designating section for enabling a user to designate an 
arbitrary image region or an arbitrary image position of an image 
without designating a contour of an object region which is 
present in the user designated image region or position; 

a specifying section for specifying the object region which 
is present in the user designated image region or position, and 
which can additionally be in a vicinity of the user designated 
image region or position, from pixel information in the user 
designated image region or position and pixel information which 
can additionally be in the vicinity of the user designated image 
region; 

a determining section for determining an image region to be 
cut out from the image, based on the specified object region; and 

a cutting section for cutting out the determined image 
region from the image. 

2. An image processing apparatus according to claim 1, wherein 
the determining section includes a section for adjusting a 

size of the image region to a prescribed size. 

3. An image processing apparatus according to claim 1, wherein 
the determining section includes a correcting section for 

entirely correcting the user designated image region or 
correcting only a part of the user designated image region. 

4. An image processing apparatus, comprising: 

a designating section for enabling a user to designate an 
arbitrary image region or an arbitrary image position of an 
image, without designating a contour of a face image region which 
is present in the user designated image region or position; 
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an analyzing section for analyzing a color distribution in 
the user designated image region or position and in a vicinity 
of the user designated region or position; 

an adjusting section for adjusting a condition for 
specifying the face image which is present in the image, 
according to a result of an analysis performed by the analyzing 
section; 

a specifying section for specifying the face image region 
which is present in the user designated image region or position, 
and which can additionally be in the vicinity of the user 
designated image region or position, based on the adjusted 
condition; 

a determining section for determining an image region to be 
cut out from the image, based on the specified face image region; 
and 

a cutting section for cutting out the determined image 
region from the image. 

5. An image processing apparatus according to claim 4, wherein 
the determining section includes a section for adjusting a 

size of the image region, using the user designated image region 
or the user designated image position as a reference. 

6. An image processing apparatus according to claim 4, wherein 
the specifying section includes: 

a section for applying noise elimination or labelling to the 
specified face image region to produce a face mask; 

a section for vertically scanning the produced face mask to 
obtain a sum of vertical differential luminance values of pixels 
in the image corresponding to the face mask to produce a 
histogram; and 

a section for detecting a central axis of a face from a 
profile of the produced histogram. 
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7. An image processing apparatus according to claim 4, wherein 
the specifying section includes: 

a section for applying noise elimination or labelling to the 
specified face image region to produce a face mask; 

a section for vertically scanning the produced face mask to 
obtain a mean luminance value of pixels in the image 
corresponding to the face mask to produce a histogram; and 

a section for detecting a vertical nose position from a 
profile of the produced histogram. 

8. An image processing apparatus according to claim 4, wherein 
the specifying section includes: 

a section for applying noise elimination or labelling to the 
specified face image region to produce a face mask; 

a section for horizontally scanning the produced face mask 
to obtain a mean luminance value of pixels in the image 
corresponding to the face mask to produce a histogram; and 

a section for detecting a vertical eye position from a 
profile of the produced histogram. 

9. An image processing apparatus according to claim 4, wherein 
the specifying section includes: 

a section for applying noise elimination or labelling to the 
specified face image region to produce a face mask; 

a section for horizontally scanning the produced face mask 
to obtain a mean luminance value of pixels in the image 
corresponding to the face mask to produce a histogram; and 

a section for detecting a vertical mouth position from a 
profile of the produced histogram. 

10. An image processing apparatus according to claim 9, wherein 
the specifying section further includes: 

a section for detecting a vertical eye position from the 
profile of the produced histogram; and 

a section for obtaining a middle position of a region 
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between the detected vertical eye position and the detected 
vertical mouth position to detect a width of the face mask at the 
middle position. 

11. An image processing apparatus according to claim 4, wherein 
the determining section includes a section for adjusting a 

position of the image region, based on the face image region, a 
central axis of a face in the face image, a vertical nose 
position of the face in the face image, a vertical eye position 
of the face in the face image, a vertical mouth position of the 
face in the face image, and a width of a face mask of the face 
image . 

12. An image processing apparatus according to claim 4, wherein 
the determining section includes a section for adjusting a 

size of the image region, based on the face image region, a 
central axis of a face in the face image, a vertical nose 
position of the face in the face image, a vertical eye position 
of the face in the face image, a vertical mouth position of the 
face in the face image, and a width of a face mask of the face 
image. 

13. An image processing apparatus according to claim 4, wherein 
the determining section includes a correcting section for 

entirely correcting the user designated image region or 
correcting only a part of the user designated image region. 
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